Safety system for autonomous operation of off-road and agricultural vehicles using machine learning for detection and identification of obstacles

ABSTRACT

A framework for safely operating autonomous machinery, such as vehicles and other heavy equipment, in an in-field or off-road environment, includes detecting, identifying, classifying and tracking objects and/or terrain characteristics from on-board sensors that capture images in front and around the autonomous machinery as it performs agricultural or other activities. The framework generates commands for navigational control of the autonomous machinery in response to perceived objects and terrain impacting safe operation. The framework processes image data and range data in multiple fields of view around the autonomous equipment to discern objects and terrain, and applies artificial intelligence techniques in one or more neural networks to accurately interpret this data for enabling such safe operation.

CROSS-REFERENCE TO RELATED PATENT APPLICATION(S)

This patent application claims priority to U.S. provisional application62/585,170, filed on Nov. 13, 2017, the contents of which areincorporated in their entirety herein. In accordance with 37 C.F.R. §1.76, a claim of priority is included in an Application Data Sheet filedconcurrently herewith.

FIELD OF THE INVENTION

The present invention relates to operation of autonomous or driverlessvehicles in an off-road and/or in-field setting. Specifically, thepresent invention relates to a system and method that applies machinelearning techniques to detect and identify objects and terrain in suchan off-road or in-field setting, and enables autonomous or driverlessvehicles to safely navigate through unpredictable operating conditions.

BACKGROUND OF THE INVENTION

Development and deployment of autonomous, driverless or unmannedvehicles and machinery have the potential to revolutionizetransportation and industrial applications of such equipment. Autonomousvehicle technology is applicable for both automotive and agriculturaluses, and in the farming industry it has great potential to increase theamount of land a farmer can work, and also significantly reduce costs.However, there are many nuances to application of autonomous vehicletechnology in an agricultural setting that make usage of such vehiclesand machinery much more difficult than in an automotive setting.

A major issue with this autonomous vehicle technology is safety, andproviding user and public confidence in the operation of equipment.Safety systems currently in use or being developed for unmanned vehiclesand machinery to-date are either specialized for automotive purposes orexceedingly expensive, and are not sufficiently accurate for full-scaledeployment, particularly in the agricultural sector where specificissues require a very high level of confidence. For example, a safetysystem used with an autonomous tractor pulling a grain cart during aharvest must be able to quickly and accurately perceive obstacles suchas people, other vehicles, fence rows, standing crop, terraces, holes,waterways, ditches, tile inlets, ponds, washouts, buildings, animals,boulders, trees, utility poles, and bales, and react accordingly toavoid mishaps. Each of these obstacles is challenging to identify with ahigh degree of accuracy.

Additionally, operating agricultural equipment and reacting accordinglywhere such obstacles have been detected and identified requires accurateon-board decision-making and responsive navigational control. However,agricultural equipment includes many different types of machines andvehicles, each with their own functions and implements for the varioustasks for which they are intended to perform, and each having adifferent profile, size, weight, shape, wheel size, stopping distance,braking system, gears, turning radius etc. Each piece of machinerytherefore has its own specific navigational nuances that make itdifficult to implement a universal or standardized approach to safeautonomous operation that can apply to any piece of agriculturalequipment.

Accordingly, there is a strong unmet need for a safety system that meetsthe substantial requirements of the agricultural marketplace and itsunique operating environments.

BRIEF SUMMARY OF THE INVENTION

The present invention is a system and method for safely operatingautonomous agricultural machinery, such as vehicles and other heavyequipment, in an in-field or off-road environment. This is provided inone or more frameworks or processes that implement various hardware andsoftware components configured to detect, identify, classify and trackobjects and/or terrain around autonomous agricultural machinery as itoperates, and generate signals and instructions for navigational controlof the autonomous agricultural machinery in response to perceivedobjects and terrain impacting safe operation. The present inventionincorporates processing of both image data and range data in multiplefields of view around the autonomous agricultural machinery to discernobjects and terrain, and applies artificial intelligence techniques inone or more trained neural networks to accurately interpret this datafor enabling such safe operation and navigational control in response todetections.

It is therefore one objective of the present invention to provide asystem and method of ensuring safe autonomous operation of machinery andvehicles in an off-road and/or in-field environment. It is anotherobjective of the present invention to provide a system and method ofensuring safe, reliable autonomous operation of machinery whileperforming agricultural tasks.

It is a further objective of the present invention to detect, identify,and classify obstacles and terrain, both in front of a vehicle and in a360° field of view around an autonomously-operated machine. It is yetanother objective of the present invention to provide a system andmethod that calculates and defines a trajectory of any objects detectedin front of vehicle and in a 360° field of view around anautonomously-operated machine. It is still a further objective of thepresent invention to apply techniques of machine learning and artificialintelligence to detect, identify, and classify obstacles and terrain,and to train one or more neural networks or other artificialintelligence tools on objects and terrain to improve performance infurther instantiations of such a safety framework.

It is still a further objective of the present invention to provide asafety system that perceives people, other vehicles, terrain, and otherin-field objects as obstacles, and determine an operational state ofautonomous field equipment in response thereto. It is yet anotherobjective of the present invention to generate one or more signals for anavigation controller configured with autonomous field equipment forsafe operation of such equipment when obstacles are detected,identified, and classified. It is another objective of the presentinvention to provide a safety system that is capable of being applied toany piece of agricultural machinery to enable its autonomous operation.

Other objects, embodiments, features, and advantages of the presentinvention will become apparent from the following description of theembodiments, which illustrate, by way of example, principles of theinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and together with the description, serve to explain theprinciples of the invention.

FIG. 1 is a system architecture diagram illustrating components in asafety framework for autonomous operation of agricultural equipmentaccording to one embodiment of the present invention;

FIG. 2 is a flowchart of steps in a process for implementing the safetyframework for autonomous operation of agricultural equipment accordingto one embodiment of the present invention;

FIG. 3 is a general block diagram of hardware components in the safetyframework for autonomous operation of agricultural equipment accordingto one embodiment of the present invention; and

FIG. 4 is an illustration of exemplary fields of view of componentscapturing input data in the safety framework for autonomous operation ofagricultural equipment according to one embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description of the present invention, reference is madeto the exemplary embodiments illustrating the principles of the presentinvention and how it is practiced. Other embodiments will be utilized topractice the present invention and structural and functional changeswill be made thereto without departing from the scope of the presentinvention.

The present invention provides an approach for ensuring safe operationof autonomous agricultural machinery, such as driverless vehicles andother heavy equipment, in an in-field or off-road environment. FIG. 1 isa system architecture diagram for a safety framework 100 for ensuringreliable operation of autonomous agricultural machinery 102. The safetyframework 100 is performed within, and is comprised of, one or moresystems and/or methods that includes several components, each of whichdefine distinct activities and functions required to process and analyzeinput data 110 from multiple types of sensors associated with suchdriverless vehicles and machinery, to recognize either or both ofobjects 104 or terrain characteristics 106 that may affect anoperational state of the autonomous agricultural machinery 102. Thesafety system 100 generates output data 140 that is used, in oneembodiment, to provide navigational control 150 for autonomousagricultural machinery 102, and provide one or more signals or commandsfor remote operation of such autonomous agricultural machinery 102 in asafe manner.

It is to be understood that the safety framework 100 may be utilizedwith any type of agricultural equipment, such as for example tractors,plows, combines, harvesters, tillers, grain carts, irrigation systemssuch as sprinklers and for any type of agricultural activity for whichautonomous operation may be implemented. Therefore, the presentspecification and invention are not to be limited to any type of machineor activity specifically referenced herein. Similarly, the safetyframework 100 may be utilized with any type of off-road vehicle ormachine, regardless of the industrial or commercial application thereof.

The safety framework 100 performs these functions by ingesting,retrieving, requesting, receiving, acquiring or otherwise obtaininginput data 110 from multiple sensors that have been configured andinitialized to observe one or more fields of view 107 around autonomousagricultural machinery 102 as it operates in a field 108. As notedfurther herein, many types of sensors may be utilized, and input data110 may be collected from either on-board sensing systems or from one ormore external of third-party sources.

The input data 110 includes images collected from at least one RGB(3-color) camera 111, which may further include a camera 112 configuredfor a forward-facing field of view 104, and a camera or system ofcameras 113 configured for a 360° degree field of view 107 around theautonomous agricultural machinery 102. The input data 110 also includesimages collected from a thermographic camera 114. Each of these cameras112, 113 and 114 may have different fields of view 107, at differentdistances relative to the autonomous agricultural machinery 102. Inputdata 110 obtained from cameras 111 may be in either raw or processedform, and therefore on-board sensing systems may include algorithms andhardware configured to process camera images for the safety framework100.

The input data 110 also includes information obtained from reflectedsignals from radio or other waves obtained from one or more rangingsystems 115. Many different types of ranging systems 115 arecontemplated, and may include ground penetrating radar 116, LiDAR 117,sonar 161, ultrasonic 162, time of flight 163, and any other rangingsystems capable of analyzing a field of view 107 around autonomousagricultural machinery 102. Each of these ranging systems 115 emitswaves in a defined field of view 107 relative to the autonomousagricultural machinery 102, and signals reflected back are utilized toidentify spatial attributes of any obstacles in the field of view 107.As with input data 110 obtained from cameras 111, information fromranging systems 115 may be in either raw or processed form, such thaton-board sensors may include algorithms and hardware capable ofprocessing such input data 110 for follow-on usage.

Input data 110 may also include GPS data 118 that enables the safetyframework 100 to correlate known obstacles with those that are detected,identified, classified and tracked in the present invention. Such GPSdata 118 enables GPS receivers to determine positional coordinatesand/or boundaries of obstacles and terrain, as well as boundaries of thefield 108 itself within which the autonomous agricultural machinery 102is being operated. This allows the safety framework 100 to apply one ormore georeferencing tags to mark known obstacles or terrain for the oneor more artificial intelligence models 128, described further herein,used to determine what objects 104 and terrain characteristics 106 arewithin the field of view 107 for the multiple sensors providing inputdata 110.

Many other types of input data 110 are also possible for use with thesafety framework 100. For example, images 119 captured by satellitesystems may also be included, and this may be used correlate knownobstacles and terrain characteristics with those that are detected,identified, classified and tracked in the present invention. Forexample, if a body of water is captured in satellite image data 119 in aparticular field 108 in which the autonomous agricultural machinery 102is operating, information about this terrain characteristic 106 may bestored with data known to a trained neural network used to detect,identify, and classify such a terrain characteristic 106, as well as toconfirm its presence in the field 108 when the multiple sensors capturepixel and spatial data that matches information representing this bodyof water.

The input data 110 is applied to a plurality of data processing modules121 within a computing environment 120 that also includes one or moreprocessors 122 and a plurality of software and hardware components. Theone or more processors 122 and plurality of software and hardwarecomponents are configured to execute program instructions or routines toperform the functions of the safety framework 100 described herein, andembodied by the plurality of data processing modules 121.

The plurality of data processing modules 121 in computing environment120 include a data initialization component 124, which is configured toinitiate collection of input data 110 from the multiple sensors andperform the ingest, retrieval, request, reception, acquisition orobtaining of input data 110. The initialization component 124 may alsobe utilized to configure the fields of view 107 of each sensorcollecting input data 110, as fields of view 107 may be definable basedon characteristics such as weather conditions being experienced orexpected in the field in which autonomous agricultural machinery 102 isoperating, the type and configuration of machinery being operated,knowledge of particular obstacles or terrain therein, and any otherlocalized or specific operating conditions that may impact each field ofview 107 and the operation of the autonomous agricultural machinery 102.

The plurality of data processing modules 121 may also include an imageand wave processing component 126, which analyzes the input data 110 toperform obstacle and terrain recognition 130. This is performed byanalyzing images captured by the multiple cameras 112, 113 and 114, andby analyzing reflected signals from radio or other waves emitted by theranging system(s) 115. The image and wave processing component 126performs a pixel analysis 131 on images from the multiple cameras 112,113 and 114, by looking for pixel attributes representing shape,brightness, color, edges, and groupings, (and other pixel attributes,such as variations in pixel intensity across an image, and across RGBchannels) that resemble known image characteristics of objects for whichthe one or more neural networks 137 have been trained. The image andwave processing component 126 also translates spatial attributes suchrange, range-rate, reflectivity and bearing 132 from the reflectedsignals from radio or other waves emitted by the ranging system(s) 115,to calculate distance, velocity and direction of the objects identifiedfrom the input data 110. This information is used to perform anidentification and classification 133 of the objects 104 and terrain106, as well as the movement and trajectory of objects 106.Geo-referencing tags 135 may also be applied to correlate objects 104and terrain 106 with known items from GPS data 118 or from priorinstantiations of the use of neural networks 137 and/or other artificialintelligence models 128 to perform the obstacle and terrain recognition130, or to mark positions of objects 104 and terrain characteristics 106identified as the autonomous agricultural machinery 102 performs itactivities.

It should be noted that the processing of input data 110, and theexecution of navigational control navigational control that isresponsive to obstacle and terrain recognition 130, occurs in real-time.It is therefore to be understood that there is no (or negligible)latency in the performance of the safety framework 100 and the variousdata processing functions described herein.

The safety framework 100 includes, as noted above, one or more layers ofartificial intelligence models 128 that are applied to assist the imageand wave processing component 126 in obstacle and terrain recognition130. The artificial intelligence portion of the present inventionincludes, and trains, one or more convolutional neural networks 137which identify, classify and track objects 104 and terraincharacteristics 106.

Use of artificial intelligence 128 operates in the safety framework 100by applying the input data 110 to the one or more neural networks 137,which receive camera data and ranging data in their various formatsthrough input layers, and then processes that incoming informationthrough a plurality of hidden layers. The one or more neural networks137 look for pixel attributes representing shape, brightness andgroupings that resemble image characteristics for which they weretrained on, and once a match is identified, the one or more neuralnetworks 137 output what has been identified, together with aprobability. For example, where a truck drives into the RGB camera'sfield of view 107, the one or more neural networks 137 may generate datain the form of (Truck)(90%). Applying such an approach to obstacledetection with a probability allows for simple filtering of falsepositives once baseline accuracy is known. Using a pre-trained neuralnetwork(s) 137, the safety framework 100 can evaluate sensor data andprovide a relatively quick solution to begin training itself further.

The image and wave processing component 126 produces output data 140that is indicative of whether an object 104 or terrain characteristic106 has been recognized that requires changing or altering theoperational state of autonomous agricultural machinery 102, or someother instruction 144 or command thereto. The output data 140 may beused to calculate a drivable pathway 142 given the object 104 or terraincharacteristic 106 recognized, and this information (or otherinstruction 144 or command) may be provided to the autonomousagricultural machinery 102 to effect navigational control 150 as theequipment moves through its intended setting. This may include a commandfor steering control 151, a stop or brake command 152, a speed controlcommand 153, and gear or mode selection 154.

Additionally, output data 140 may be provided as an input to performpath planning, by extrapolating the position of the detected object 104or terrain characteristic 106 in a mapping function 155, and calculatinga new route 156 to avoid such obstacles. In such a path planningembodiment, output data 140 may be georeferencing data, together with atrigger, and the command for navigational control 150 is to re-plan anew route 156, or the new route 156 itself. Also, a command or data fora mapping function 155 itself may also be provided. For example,depending on the type of object 104 or terrain characteristic 106detected, the obstacle may be updated either temporarily or permanently,until the obstacle is in the field of view 107. In such an example, astatic object 104 such as a pole, or non-traversable terrain 106, mayproduce an update to the mapping function 155, and the terraincharacteristic 106 may be marked as an exclusion or no-go zone.Similarly, a dynamic object 104 such as a person may require only atemporary update to the mapping function 155.

Regardless, is to be understood that many other commands fornavigational control derived from the output data 140 are also possibleand within the scope of the present invention, and therefore thisdisclosure is not to be limited to any instruction 144 or commandspecifically delineated herein.

A calculated drivable pathway 142 may take many factors into account,and use other types of input data 110, to respond to detected andidentified objects 104 and terrain characteristics 106, and providesignals for a navigational controller 150 to take action to ensuresafety in the present invention. For example, the safety framework 100may evaluate GPS data 118 to continually identify a position and aheading of the autonomous agricultural machinery 102 as it operatesthrough a field 108. Additionally, path planning in calculating adrivable pathway and navigational control in response thereto may takeinto account operational characteristics of the particular equipment inuse, such as its physical dimensions and the type of nature ofimplements configured thereon, as well as the turning radius, currentspeed, weather conditions, etc. Further, as noted herein, outer andinner field boundaries (and positional coordinates thereof) that forexample define exclusion zones and other field limitations must also beaccounted for.

The safety framework 100 of the present invention uses a plurality ofsensors so that an object 104 and terrain 106 may be identified andlocated using more than one source, both to improve accuracy and toaccount for operating conditions where reliability of sources may beimpaired. As one skilled in the art will readily appreciate,environmental factors may affect the ability of the safety framework 100to identify and locate an object 104 and terrain 106, as images andreflected radio or other signals in the fields of view 107 may not besufficient for the neural network(s) 137 to properly perform. Forexample, when a level of light is relatively low, an RGB camera 111 maynot generate enough data to allow a neural network 137 to identify anobject photographed by that sensor. Similarly, in settings where theenvironment and the objects within it have substantially the sametemperature, a neural network 137 utilizing data from thermographiccamera 114 may not be able to identify an object. However, thecombination of an RGB camera 111 and a thermographic camera 114 greatlyimproves the ability for the safety framework 100 to accurately detect,identify and classify an object 104. For example, where autonomousagricultural machinery 102 utilizing the safety framework 1000 isdeployed at night, and an object 104 is in the field of view 107 of theRGB camera 111 and the thermographic camera 114, the neural networks 137may be unable to identify or classify the object 104 based on dataobtained from the RGB camera 111. However, the thermographic camera 114may provide enough information to allow the neural network(s) 137 todetect the presence of the object 104 and then further classify it.

Similarly, if the safety framework 100 is deployed in a relatively warmlight environment, for example, a farm field on a warm summer day, thethermographic camera 114 may not be able to generate enough data for theneural network 137 to identify an object 104 within its field of view107. However, if there is enough light in such an operational setting,an identification may be made from the data collected by the RGB camera111.

Navigational control 150 of the autonomous agricultural machinery 102may depend vary based on multiple factors, such as for example the typeof the identified object 104 or terrain characteristic 106, and thedistance the object 104 or terrain characteristic 106 is from theautonomous agricultural machinery 102, and its movement. For example,the object 104 may be identified as a person 50 feet away. In response,the autonomous agricultural machinery 102 may slow its speed in order togive the person an opportunity to avoid the vehicle. If the person doesnot move, the autonomous agricultural machinery 102 may slow to a lower(or predetermined) speed, by either braking or lowering to selectedgear, as the autonomous agricultural machinery 102 approaches theperson, or may turn to follow an alternate pathway in the event it isdetermined the person has not moved. The autonomous agriculturalmachinery 102 may also be instructed to stop if the person has not movedfrom the approaching autonomous agricultural machinery 102, and may alsobe configured to emit a loud noise to warn the person of an approachingvehicle. In the alternative, if the object 104 is identified as acoyote, the autonomous agricultural machinery 102 may simply progresswithout changing its course or speed, or emit a warning sound orhigh-frequency signal. As yet another alternative, if the object 104cannot be sufficiently identified, the navigational controller 150 maystop the autonomous agricultural machinery 102 and contact the operatorto alert the operator of the object 104, and allow for a non-autonomousdetermination of a course of action that should be taken. In this latterembodiment, the navigational controller 150 may cause a digital image ofthe obstacle taken by a camera to be sent wirelessly to the operator forfurther analysis.

It is to be understood that the plurality of sensors that capture inputdata 110 may be both configured on-board autonomous agriculturalmachinery 102, so as to collect input data 110 as the autonomousagricultural machinery 102 operates, or otherwise associated with suchautonomous agricultural machinery 102 so that sensors need not bephysically coupled to such machinery 102. For example, where the safetyframework 100 of the present invention includes satellite data 119 inits processing, such data 119 may be ingested, received, acquired, orotherwise obtained from third party of external sources. Additionally,it is also contemplated and within the scope of the present inventionthat the safety framework 100 may utilize data 110 collected by othervehicles, driverless or otherwise, operating in the same field as theautonomous agricultural machinery 102, either at the same time or atother relevant temporal instances. For example, one piece of machinerymay capture a body of water present in a field at a prior time period onthe same day, and this may be used by the present invention to make adetermination of whether an object 104 or terrain 106 later identifiedrequires a change in operational state or navigational control.

As noted above, machine learning is used in the safety framework 100 toassociate and compare information in the various types of input data 110and identify attributes in such input data 110 to produce identificationand classification of objects 104 and terrain characteristics 106, andto track movement of objects 104. This information is ultimately used togenerate output data 140, which enables the safety framework 100 tocalculate a drivable pathway 142 for the autonomous agriculturalmachinery 102 and generate instructions 144 for navigational control 150thereof. As part of the processing performed in the safety framework100, the one or more neural networks 137 may be configured to developrelationships among and between the various types of input data 110 toperform the correlations and matching used to formulate obstacle andterrain recognition 130, which is used to determine whether the safetyframework 100 needs to take action to manipulate and control theautonomous agricultural machinery 102 in response to the unexpectedpresence of an object 104 or unknown terrain characteristic 106.

The present invention contemplates that temporal and spatial attributesin the various types of input data 110 may be identified and developedin such a combined analysis by training the one or more layers ofartificial intelligence 128 to continually analyze input data 110, tobuild a comprehensive dataset that can be used to make far-reachingimprovements to how objects 104 and terrain 106 are determined asautonomous agricultural machinery 102 operates in a field 108. Forinstance, the one or more layers of artificial intelligence 128 can beapplied to an adequately-sized dataset to draw automatic associationsand identify attributes in pixels, effectively yielding a customizedmodel for that can identify commonly-encountered objects or terrain in aparticular field. As more and more data are accumulated, the informationcan be sub-sampled, the one or more neural networks 137 retrained, andthe results tested against independent data representing known objectsand terrain, in an effort to further improve obstacle and terrainrecognition 130 in the safety framework 100. Further, this informationmay be used to identify which factors are particularly important orunimportant in associating temporal and spatial attributes and othercharacteristics when identifying and classifying objects and terrain,and tracking movement of objects, thus helping to improve the accuracyand speed of the safety framework 100 over time.

The present invention contemplates that many different types ofartificial intelligence may be employed within the scope thereof, andtherefore, the artificial intelligence component 128 and modelscomprised thereof may include one or more of such types of artificialintelligence. The artificial intelligence component 128 may applytechniques that include, but are not limited to, k-nearest neighbor(KNN), logistic regression, support vector machines or networks (SVM),and one or more neural networks 137 as noted above. It is to be furtherunderstood that any type of neural network 137 may be used, and thesafety framework 100 is not to be limited to any one type of neuralnetwork 137 specifically referred to herein. Regardless, the use ofartificial intelligence in the safety framework 100 of the presentinvention enhances the utility of obstacle and terrain recognition 130by automatically and heuristically identifying pixel attributes such asshapes, brightness and groupings, using mathematical relationships orother means for constructing relationships between data points ininformation obtained from cameras 111 and 114, and ranging systems 115,to accurately identify, classify and track objects 104 and terrain 106,where applicable. For example, where pixel characteristics known to berelated to a particular object or terrain characteristic are known andanalyzed with the actual objects/terrain in real-world situations,artificial intelligence techniques 128 are used to ‘train’ or constructa neural network 137 that relates the more readily-available pixelcharacteristics to the ultimate outcomes, without any specific a prioriknowledge as to the form of those attributes.

The neural network(s) 137 in the present invention may be comprised of aconvolutional neural network (CNN). Other types of neural networks arealso contemplated, such as a fully convolutional neural network (FCN),or a Recurrent Neural Network (RNN), and are within the scope of thepresent invention. Regardless, the present invention applies neuralnetworks 137 that are capable of utilizing image data collected from acamera 111 or thermal imaging device 114 to identify an object 104 orterrain 106. Such neural networks 137 are easily trained to recognizepeople, vehicles, animals, buildings, signs, etc. Neural networks arewell known in the art and many commercial versions are available to thepublic. It is to be understood that the present invention is not to belimited to any particular neural network referred to herein.

FIG. 2 is a flowchart illustrating a process 200 for performing thesafety framework 100 of the present invention. The process 200 begins atstep 210 by initializing sensor systems on, or associated with,autonomous agricultural machinery 102, for example where agriculturalapplications in performing field activities are commenced usingdriverless vehicles and equipment. The sensor systems at step 210 areactivated and begin the process of continually observing the definedfields of view 107, and at step 220 this input data 110 from cameras 111and 114 and ranging systems 115 is collected as autonomous agriculturalmachinery 102 operates in a selected environment. At step 230, theprocess 200 analyzes pixels from images captured by the cameras 111 and114, and translates signals reflected from waves emitted by the rangingsystems 115.

At step 240, the process 200 applies one or more trained neural networks137 to perform recognition 130 of objects 104 and terraincharacteristics 106 as described in detail above. At step 250, the oneor more neural networks 137 identify and classify certain objects 104and terrain 106 in camera images, as well as determine spatialattributes such as distance and position to locate objects 104 andterrain 106, and to determine movement at least in terms of velocity anddirection to track objects 106 from both image and ranging data. Theneural networks 137 are also constantly being trained to “learn” how todiscern and distinguish items encountered by the autonomous agriculturalmachinery 102 as input data 110 is collected and as objects 104 andterrain 106 are recognized, characterized, and confirmed, at step 252.At step 260, the present invention calculates a trajectory of theobjects 104 to further characterize the object 104 and help determinethe operational state of the autonomous agricultural machinery 102 inresponse thereto. At steps 250, 252, and 260 therefore, the process 200continually trains one or more artificial intelligence models to improveidentification of images obtained using cameras 111 and 114 and rangingsystems 115, and improving the ability to perform depth relation andtrack directional movement and speed, and other identification andlocation characterizations, that help to accurately determine objects104 and terrain 106 in a field. As noted above, many types of outputsare possible from the safety framework 100. In one such possible output,in step 260, the process 200 may perform an update to a mapping function155 once obstacles such as objects 104 and terrain characteristics 106have been detected, identified and classified.

At step 270, the process 200 applies the information obtained regardingany objects 104 or terrain characteristics 106, and calculates adrivable pathway to reach an intended waypoint or endpoint thatacknowledges the in-field obstacle. At step 280, the process thendetermines whether an operational state of the autonomous agriculturalmachinery 102 must be altered in response to the calculated drivablepathway 142. This may include determining whether an object 104 orterrain characteristic 106 is an in-field obstacle that requires anadjustment of the path or operation of the autonomous agriculturalmachinery 102. For example, and as noted above, a drivable pathwayaround a coyote may be calculated, but the safety framework 100 maydetermine to proceed along the current pathway, with or without anadjustment to some operational state such as increasing or decreasingspeed.

At step 290, the process 200 generates output data 140 that may includeinstructions to control navigation of the autonomous agriculturalequipment in response to the calculated drivable pathway, and/orotherwise in response to a change the operational state of theautonomous agricultural equipment, where an object 104 or terraincharacteristic 106 requires than an action be taken.

It is to be understood that autonomous operation of vehicles andmachinery for agricultural applications or in other field/off-roadenvironments requires extensive configuration for safe and accurateperformance, such as field setup and location mapping to ready thevarious hardware and software elements associated with agriculturalequipment for driverless activity. This may include defining fieldboundaries and one or more way or destination points that serve aspositions in a field where such vehicles and machinery are required tooperate to perform autonomous agricultural tasks. One aspect of ensuringaccurate and safe performance in autonomous operation of vehicles andmachinery in agricultural applications is the usage of boundaries andother way paints as a safety mechanism, and the present inventionincludes software configured such that the autonomous agriculturalmachinery 102 may only operate within the pre-established boundaries ofthe field 108. For example, an outer boundary may be ported into acontroller platform on board the autonomous agricultural machinery 102,either from another “precision” agricultural device, or created by auser from satellite imagery 119. If the autonomous agriculturalmachinery 102 projects an autonomous waypoint path such that any pointalong the waypoint path is outside of a pre-set boundary, the autonomousagricultural machinery 102 will issue a warning to the operator and willfail to start. Internal boundaries can also be created as operation ofthe autonomous agricultural machinery 102 progresses by a user such asthe combine operator. Inner boundaries then become exclusion zones thatthe autonomous agricultural machinery 102 is to avoid. In this manner,calculation of a drivable pathway 142 in the present invention takesinto account pre-set as well as in-operation boundaries and waypoints,such as field boundaries and inner boundaries defining exclusion zonesto be avoided, in addition to objects 104 and other terraincharacteristics 106 requiring changes in operational states such assteering 151, stopping and braking 152, increasing or decreasing speed153, gear/mode selection 154, and other manipulations.

FIG. 3 is a generalized block diagram of an exemplary hardwareconfiguration 300 for the safety framework 100 for autonomous operationof agricultural machinery 102. The exemplary hardware configuration 300includes a plurality of sensors 310 and 330, which as discussed hereinmay include a forward-facing RGB camera 112, a camera or camera systemsconfigured for a 360° view 113, and a thermographic camera 114. Sensors330 may include a ranging system 115, such as ground penetrating radar116 or any other kind of range or radar system, as noted above.

The exemplary hardware configuration 300 also includes an on-boardcontroller 320 that has a graphics processing unit (GPU) and a carrierboard implementing such a GPU, and a navigational controller 340. Theon-board controller 320 may include and utilize one or more softwarecomponents performing algorithms that filter and fuse sensor data, andapply techniques of artificial intelligence to analyze such sensor datato perform the image and wave processing described herein. Thenavigational controller 340 may similar include and utilize one or moresoftware components performing algorithms that enable navigation of theagricultural equipment as it operates in its intended setting for theperformance of autonomous tasks and activities.

Several input/output (I/O) configurations provide connectivity betweenthese elements, such as a serial CAN (Controller Area Network) bus 360which may be utilized to connect the ranging sensor 330 to the on-boardcontroller 320 and provide power thereto, and one or more physical/wiredconnections 350 such as Gigabit Multimedia Serial Link (GMSL), USB 3.0,and a serializer/de-serializer (SerDes) that connect the camera sensors310 to the on-board controller (and also provide power thereto). It isto be understood however than many types of configurations, either wiredor wireless, are possible for connecting the plurality of sensorsconfigured on autonomous agricultural machinery 102 to the controller(s)thereon, and are within the scope of the present invention, and thesafety framework 100 is therefore not intended to be limited to any oneconfiguration shown or described herein. Similarly, Ethernet, Wi-Fi orBluetooth® (or another means of connectivity) may be utilized to linkthe on-board controller 320 with the navigational controller 340, andtherefore it is to be understood that such a connection may also beeither wired or wireless and may take any form that enables suchelements to effectively communicate information.

In one exemplary physical embodiment, the GPR sensing unit 330 ismounted on the front of a vehicle above a weight rack, and connected tothe GPU with a CAN bus cable which also provides power to therange/radar components. The thermal, RGB and 360-degree cameras 310 aremounted below and in front of the vehicle cab's centralized GPS mountinglocation to provide the best field of view 107 for the cameras 111 and114. These imaging sensors 310 are powered via physical connections 350,such as for example USB 3.0, GMSL, and Ser/Des to the GPU processor 320.The GPU processor 320 itself may be mounted next to the navigationcontroller 340 and interfaced over Ethernet, Wi-Fi or Bluetooth® asnoted above.

FIG. 4 is an illustration 400 of exemplary fields of view 107 for sensorcomponents capturing input data 110 in the present invention. Thesefields of view 107 may be customizable by owners or operators ofautonomous agricultural machinery 102, for example using a remotesupport tool as noted further herein. Fields of view 107 may bechangeable for many different reasons, such as for example the intendeduse of the agricultural machinery 102, the type of machinery 102 onwhich they are mounted, for various weather conditions, and foroperational limitations of the sensors themselves.

In the illustration 400 of FIG. 4, each of the sensors 410, 420, 430 and440 have different fields of view 107 and each provide a distinctiveview of the area around autonomous agricultural machinery 102 thecollectively represent a comprehensive ability to detect objects 104 orterrain 106. For example, the 360° camera 410 has a field of view 113that extends in a radius around the autonomous agricultural machinery102 (not shown), allowing the camera 410 to see all around thedriverless vehicle. This enables detection of obstacles in a 360° areanear or beside a driverless machine, at a range 50% greater than thewidth of the machine itself. The thermographic camera 420 has a field ofview 114, extending in a forward-facing configuration to capture thermalimages further than that of the 360° camera's capabilities. Another RGBcamera 430 has a field of view 112 that extends even further inforward-facing direction beyond that of the other two cameras. Finally,the ranging system 440 has field of view 116 that is narrower but longerthan that of the other sensing systems. Together, the fields of view 107in FIG. 4 are able to detect obstacles at a range of at least 100 metersin front of the autonomous agricultural machinery 102.

The safety framework 100 may also include a remote stop system thatutilizes a mesh network topology to communicate between emergency stopdevices and the autonomous agricultural machinery 102, either inconjunction with an output from the navigational controller 150 orseparately in response to a recognized object 104 or terraincharacteristic 106. The remote stop system is integrated into thedriverless vehicle's control interface device, and when activated,broadcasts a multicast emergency stop message throughout the distributedmesh network. The mesh radio integrated into the vehicle's controlinterface device receives the message and when received, initiates theemergency stop procedure. The emergency stop procedure is performedoutside the application layer and works at the physical layer of theinterface device. This serves as a redundant safety protocol thatassures that if a catastrophic software defect occurs in the autonomousvehicle application, the safety stop procedure can still be performed.The mesh network topology allows for messages to hop from one line ofsight device to another allowing for a message to hop across thetopology to reach non-line-of-sight nodes in the network. This acts tonot only provide a way for everyone to stop the autonomous vehicle inthe field, but also works to increase the node density of the networkand increase the remote stop range and bandwidth.

The present invention may also include a support tool that is configuredto allow access for configuration of the plurality of sensors, fields ofview 107, and navigational decision-making in response to recognition130 of objects 104 and terrain characteristics 106 in the safetyframework 100 of the present invention.

The support tool may also enable a user to input and/or selectoperational variables for conducting operations with the autonomousagricultural machinery 102 that are related to ensuring its safe andaccurate job performance. For example, operational field boundaries canbe input or selected, as well as attributes (such as GPS coordinatesand, boundaries, and sizes) of field conditions, such as the presence ofobjects 104 or terrain characteristics 106, that are already known tothe user.

The support tool may further include a function enabling a user overridethat overrides automatic navigational control of the autonomousagricultural machinery 102. Such a user override allows a user toinstruct the safety framework 100 to ignore a detected object 104 orterrain characteristic 106 and proceed with performance of theautonomous agricultural activity. The support tool may further beconfigured to generate recommendations, maps, or reports as output data,such as for example a report describing navigational actions taken inresponse to objects 104 or terrain 106 detected, types of objects 104and terrain characteristics 106 detected, and locations within aparticular field 108 of interest.

The support tool may be configured for visual representation to users,for example on a graphical user interface, and users may be able toconfigure settings for, and view various aspects of, safety framework100 using a display on such graphical user interfaces, and/or viaweb-based or application-based modules. Tools and pull-down menus onsuch a display (or in web-based or application-based modules) may alsobe provided to customize the sensors providing the input data 110, aswell as to modify the fields of view 107. In addition to desktop,laptop, and mainframe computing systems, users may access the supporttool using applications resident on mobile telephony, tablet, orwearable computing devices.

The systems and methods of the present invention may be implemented inmany different computing environments. For example, the safety framework100 may be implemented in conjunction with a special purpose computer, aprogrammed microprocessor or microcontroller and peripheral integratedcircuit element(s), an ASIC or other integrated circuit, a digitalsignal processor, electronic or logic circuitry such as discrete elementcircuit, a programmable logic device or gate array such as a PLD, PLA,FPGA, PAL, and any comparable means. In general, any means ofimplementing the methodology illustrated herein can be used to implementthe various aspects of the present invention. Exemplary hardware thatcan be used for the present invention includes computers, handhelddevices, telephones (e.g., cellular, Internet enabled, digital, analog,hybrids, and others), and other such hardware. Some of these devicesinclude processors (e.g., a single or multiple microprocessors), memory,nonvolatile storage, input devices, and output devices. Furthermore,alternative software implementations including, but not limited to,distributed processing, parallel processing, or virtual machineprocessing can also be configured to perform the methods describedherein.

The systems and methods of the present invention may also be partiallyimplemented in software that can be stored on a storage medium, executedon programmed general-purpose computer with the cooperation of acontroller and memory, a special purpose computer, a microprocessor, orthe like. In these instances, the systems and methods of this inventioncan be implemented as a program embedded on personal computer such as anapplet, JAVA® or CGI script, as a resource residing on a server orcomputer workstation, as a routine embedded in a dedicated measurementsystem, system component, or the like. The system can also beimplemented by physically incorporating the system and/or method into asoftware and/or hardware system.

Additionally, the data processing functions disclosed herein may beperformed by one or more program instructions stored in or executed bysuch memory, and further may be performed by one or more modulesconfigured to carry out those program instructions. Modules are intendedto refer to any known or later developed hardware, software, firmware,artificial intelligence, fuzzy logic, expert system or combination ofhardware and software that is capable of performing the data processingfunctionality described herein.

The foregoing descriptions of embodiments of the present invention havebeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Accordingly, many alterations, modifications andvariations are possible in light of the above teachings, may be made bythose having ordinary skill in the art without departing from the spiritand scope of the invention. It is therefore intended that the scope ofthe invention be limited not by this detailed description. For example,notwithstanding the fact that the elements of a claim are set forthbelow in a certain combination, it must be expressly understood that theinvention includes other combinations of fewer, more or differentelements, which are disclosed in above even when not initially claimedin such combinations.

The words used in this specification to describe the invention and itsvarious embodiments are to be understood not only in the sense of theircommonly defined meanings, but to include by special definition in thisspecification structure, material or acts beyond the scope of thecommonly defined meanings. Thus if an element can be understood in thecontext of this specification as including more than one meaning, thenits use in a claim must be understood as being generic to all possiblemeanings supported by the specification and by the word itself.

The definitions of the words or elements of the following claims are,therefore, defined in this specification to include not only thecombination of elements which are literally set forth, but allequivalent structure, material or acts for performing substantially thesame function in substantially the same way to obtain substantially thesame result. In this sense it is therefore contemplated that anequivalent substitution of two or more elements may be made for any oneof the elements in the claims below or that a single element may besubstituted for two or more elements in a claim. Although elements maybe described above as acting in certain combinations and even initiallyclaimed as such, it is to be expressly understood that one or moreelements from a claimed combination can in some cases be excised fromthe combination and that the claimed combination may be directed to asub-combination or variation of a sub-combination.

Insubstantial changes from the claimed subject matter as viewed by aperson with ordinary skill in the art, now known or later devised, areexpressly contemplated as being equivalently within the scope of theclaims. Therefore, obvious substitutions now or later known to one withordinary skill in the art are defined to be within the scope of thedefined elements.

The claims are thus to be understood to include what is specificallyillustrated and described above, what is conceptually equivalent, whatcan be obviously substituted and also what essentially incorporates theessential idea of the invention.

The invention claimed is:
 1. A method, comprising: initializing inputdata collected from a plurality of sensors mounted on autonomousoff-road machinery performing one or more agricultural activities in afield, the plurality of sensors including a ranging system, and aplurality of cameras configured to capture images, in multiple fields ofview from the autonomous off-road machinery; analyzing the input datawithin a computing environment in one or more data processing modulesexecuted in conjunction with at least one specifically-configuredprocessor, the one or more data processing modules configured torecognize objects and terrain characteristics around the autonomousoff-road machinery, by analyzing one or more attributes of pixels inimages captured by the plurality of cameras, and analyzing one or moreattributes captured from signals reflected from waves communicated bythe ranging system, to detect both a presence and a location of theobjects and terrain characteristics in the multiple fields of view,applying one or more neural networks to identify and classify theobjects and terrain characteristics in the multiple fields of view, bymatching image characteristics that resemble one or more of pixelshapes, pixel brightness and pixel groupings used to train the one ormore neural networks, tracking movement of objects identified andclassified by the one or more neural networks, to determine one or moreof distance, direction, and velocity of the objects, by matching spatialattributes that are similar to characteristics of objects used to trainthe one or more neural networks, and calculating a drivable pathway inresponse to the presence and location of the objects and terraincharacteristics in the multiple fields of view, and movement of theobjects; and generating one or more instructions to control navigationof the autonomous off-road machinery in a performance of the one or moreagricultural activities in the field in response to the drivablepathway.
 2. The method of claim 1, wherein the generating one or moreinstructions to control navigation of the autonomous off-road machineryin a performance of the one or more agricultural activities in the fieldin response to the drivable pathway further comprises generating one ormore commands to change an operational state of the autonomous off-roadmachinery.
 3. The method of claim 2, wherein the change in operationalstate includes one or more of steering, stopping, braking, increasingspeed, decreasing speed, selecting one or more gears, shifting the oneor more gears, and selecting an operational mode of the autonomousoff-road machinery.
 4. The method of claim 1, wherein the drivablepathway is responsive to heading and position data of the autonomousoff-road machinery, and to operational characteristics that include oneor more of turning radius capability, implemented attachments, andintended usage of the autonomous off-road machinery.
 5. The method ofclaim 1, further comprising applying geo-referencing techniques to tag aposition of each of the objects and terrain characteristic identifiedand classified.
 6. The method of claim 1, further comprising evaluatingGPS data to continually identify a position and a heading of theautonomous off-road machinery.
 7. The method of claim 1, wherein theplurality of cameras configured to capture images in multiple fields ofview around the autonomous off-road machinery include one or more of athermographic camera, a forward-facing camera and a camera configuredfor a 360-degree field of view.
 8. The method of claim 1, wherein theanalyzing one or more attributes of pixels further comprises identifyingpixel attributes that include at least one of shape, color, brightness,edges and groupings to calculate image characteristics representing theobjects and terrain characteristics in the multiple fields of view. 9.The method of claim 1, wherein the analyzing one or more attributescaptured from signals reflected from waves communicated by the rangingsystem further comprises evaluating range, reflectivity, and bearingdata to calculate an object's spatial attributes in the multiple fieldsof view.
 10. The method of claim 1, further comprising continuallytraining the one or more neural networks with pixel attributes of shapesin objects and terrain characteristics identified and classified, andwith spatial attributes of objects tracked.
 11. A system, comprising: acomputing environment including at least one non-transitorycomputer-readable storage medium having program instructions storedtherein and a computer processor operable to execute the programinstructions within one or more data processing modules configured torecognize objects and terrain characteristics around autonomousagricultural machinery performing one or more agricultural activities ina field, the one or more data processing modules including: a datacollection component configured to initialize input data collected froma plurality of sensors mounted on the autonomous agricultural machinery,the plurality of sensors including a ranging system, and a plurality ofcameras configured to capture images, in multiple fields of view fromthe autonomous agricultural machinery; an image and wave processingcomponent configured to analyze one or more attributes of pixels inimages captured by the plurality of cameras, and analyzing one or moreattributes captured from signals reflected from waves communicated bythe ranging system, to detect both a presence and a location of theobjects and terrain characteristics in the multiple fields of view; anartificial intelligence component configured to identify and classifythe objects and terrain characteristics in the multiple fields of view,by matching image characteristics that resemble one or more of pixelshapes, pixel colors, pixel brightness, pixel edges, and pixel groupingsused to train the one or more neural networks, and estimate a trajectoryof an object identified and classified to determine one or more ofdistance, direction, and velocity of the objects, by matching spatialattributes that are similar to characteristics of objects used to trainthe one or more neural networks; an output data component configured tocalculate a drivable pathway in response to the presence of the objectsand terrain characteristics in the multiple fields of view; and anavigational control component configured to generate one or moreinstructions for actuating the autonomous agricultural machinery for aperformance of the one or more agricultural activities in the field inresponse to the drivable pathway.
 12. The system of claim 11, whereinthe navigational control component generates one or more commands tochange an operational state of the autonomous agricultural machinery.13. The system of claim 12, wherein the change in operational stateincludes one or more of steering, stopping, braking, increasing speed,decreasing speed, selecting one or more gears, shifting the one or moregears, and selecting an operational mode of the autonomous agriculturalmachinery.
 14. The system of claim 11, wherein the drivable pathway isresponsive to heading and position data of the autonomous agriculturalmachinery, and to operational characteristics that include one or moreof turning radius capability, implemented attachments, and intendedusage of the autonomous off-road machinery.
 15. The system of claim 11,wherein the image and wave processing component applies geo-referencingtechniques to tag a position of each of the objects and terraincharacteristic identified and classified.
 16. The system of claim 11,further comprising a component configured to evaluate GPS data tocontinually identify a position and a heading of the autonomousagricultural machinery.
 17. The system of claim 11, wherein theplurality of cameras configured to capture images in multiple fields ofview around the autonomous agricultural machinery include one or more ofa thermographic camera, a forward-facing camera and a camera configuredfor a 360-degree field of view.
 18. The system of claim 11, wherein theimage and wave processing component is further configured to analyze oneor more attributes of pixels by identifying pixel attributes thatinclude at least one of shape, color, brightness, edges and groupings tocalculate image characteristics representing the objects and terraincharacteristics in the multiple fields of view.
 19. The system of claim11, wherein the image and wave processing component is furtherconfigured to analyze one or more attributes captured from signalsreflected from waves communicated by the ranging system by evaluatingrange, reflectivity, and bearing data to calculate an object's spatialattributes in the multiple fields of view.
 20. The system of claim 11,wherein the artificial intelligence component is further configured tocontinually train the one or more neural networks with pixel attributesof shapes in objects and terrain characteristics identified andclassified, and with spatial attributes of objects tracked.
 21. A methodfor ensuring safe operation of autonomous agricultural machinery,comprising: analyzing input data collected from a plurality of sensorsmounted on autonomous agricultural machinery in performing one or moreagricultural activities in a field, the plurality of sensors including aranging system, and a plurality of cameras configured to capture images,in multiple fields of view from the autonomous agricultural machinery;analyzing one or more attributes of pixels in images captured by theplurality of cameras, and analyzing one or more attributes captured fromsignals reflected from waves communicated by the ranging system, todetect both a presence and a location of objects and terraincharacteristics in the multiple fields of view; matching imagecharacteristics that resemble one or more of pixel shapes, pixel colors,pixel brightness, pixel edges, and pixel groupings in one or moretrained neural networks to identify and classify the objects and terraincharacteristics in the multiple fields of view; matching spatialattributes that are similar to characteristics of objects used to trainthe one or more neural networks to calculate a trajectory of the objectidentified and classified by the trained one or more neural networksfrom one or more of distance, direction, and velocity of the objects;calculating a drivable pathway in response to the presence of theobjects and terrain characteristics in the multiple fields of view;determining a change in an operational state of the autonomousagricultural machinery to accommodate the drivable pathway; andcontrolling navigation of the autonomous agricultural machinery in aperformance of the one or more agricultural activities in the field inresponse to the drivable pathway and to effect the change in operationalstate, by generating commands to perform one or more of steering,stopping, braking, increasing speed, decreasing speed, selecting one ormore gears, shifting the one or more gears, and selecting an operationalmode of the autonomous agricultural machinery.
 22. The method of claim21, wherein the drivable pathway is responsive to heading and positiondata of the autonomous agricultural machinery, and to operationalcharacteristics that include one or more of turning radius capability,implemented attachments, and intended usage of the autonomousagricultural machinery.
 23. The method of claim 21, further comprisingapplying geo-referencing techniques to tag a position of each of theobjects and terrain characteristic identified and classified.
 24. Themethod of claim 21, further comprising evaluating GPS data tocontinually identify a position and a heading of the autonomousagricultural machinery.
 25. The method of claim 21, wherein theplurality of cameras configured to capture images in multiple fields ofview around the autonomous agricultural machinery include one or more ofa thermographic camera, a forward-facing camera and a camera configuredfor a 360-degree field of view.
 26. The method of claim 11, wherein theanalyzing one or more attributes of pixels further comprises identifyingpixel attributes that include at least one of shape, color, brightness,edges and groupings to calculate image characteristics representing theobjects and terrain characteristics in the multiple fields of view. 27.The method of claim 21, wherein the analyzing one or more attributescaptured from signals reflected from waves communicated by the rangingsystem further comprises evaluating range, reflectivity, and bearingdata to calculate an object's spatial attributes in the multiple fieldsof view.
 28. The method of claim 1, further comprising continuallytraining the one or more neural networks with pixel attributes of shapesin objects and terrain characteristics identified and classified, andwith spatial attributes of objects tracked.